Papers with Prompting language models
Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models (2022.findings-acl)
Copied to clipboard
| Challenge: | Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. |
| Approach: | They propose to fine tune masked language models with training examples and task descriptions to reduce prompt engineering by using null prompts. |
| Outcome: | The proposed prompts can be used to improve few-shot learning by finetuning only the bias terms while updating only 0.1% of the parameters. |
A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains (2024.acl-long)
Copied to clipboard
Alon Jacovi, Yonatan Bitton, Bernd Bohnet, Jonathan Herzig, Or Honovich, Michael Tseng, Michael Collins, Roee Aharoni, Mor Geva
| Challenge: | Recent literature discusses automatic methods to evaluate reasoning to improve their correctness, but no fine-grained step-level datasets are available to enable thorough evaluation of such verification methods. |
| Approach: | They propose to benchmark automatic verifiers of complex Chain-of-Thought reasoning in open-domain question-answering settings using a dataset that includes comprehensive labels for relevance, attribution to evidence passages, and logical correctness of each reasoning step. |
| Outcome: | The proposed dataset shows that verifiers struggle at verifying reasoning chains, particularly verifying logical correctness and detecting contradictions. |
Tree Prompting: Efficient Task Adaptation without Fine-Tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained language models (LMs) are the main interface for applying them to new tasks, but their large size makes them difficult to fine-tune with gradients for specific downstream tasks. |
| Approach: | They propose to use training data to form a decision tree based on prompt-LM calls, with each prompt determined by the outcomes of previous calls. |
| Outcome: | The proposed method improves accuracy over competing methods and is competitive with fine-tuning. |